Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [X.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXXXXXXX] duration: 5.000728078s trace: https://jaeger.dev.storj.io/trace/4283e006a1d53af1 FAIL (2026-01-12 14:38:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000748848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........X...X...X...X.............X..XXX..................] duration: 34.191477422s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:38:04 UTC) orbiter warning: context deadline exceeded Duration: 34.191504644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X..X..] duration: 27.920778693s trace: https://jaeger.dev.storj.io/trace/5f5b45216158852f FAIL (2026-01-12 14:37:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.920794613s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXXXXXX] duration: 5.001674123s trace: https://jaeger.dev.storj.io/trace/64255ca3d0b0dc07 FAIL (2026-01-12 14:37:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001695424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.80326442s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.803304376s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.636838347s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:37:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.636867613s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X..X.] duration: 27.557236464s trace: https://jaeger.dev.storj.io/trace/5f5b45216158852f FAIL (2026-01-12 14:37:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.557251133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.747691819s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.747706006s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X..X] duration: 30.002365347s trace: https://jaeger.dev.storj.io/trace/5f5b45216158852f FAIL (2026-01-12 14:36:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002376119s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 13.863770777s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:36:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 13.863799352s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXXXXX] duration: 5.001526946s trace: https://jaeger.dev.storj.io/trace/64fffc8f74ae8b6a FAIL (2026-01-12 14:36:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001553696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X...X...X...X.............X..XXX.................] duration: 17.613828722s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:36:46 UTC) orbiter warning: context deadline exceeded Duration: 17.613851196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.82068319s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.820712817s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X..] duration: 29.43988103s trace: https://jaeger.dev.storj.io/trace/29e2e68844fc3c8f FAIL (2026-01-12 14:36:26 UTC) orbiter warning: context deadline exceeded Duration: 29.439904764s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXXXX] duration: 5.000559517s trace: https://jaeger.dev.storj.io/trace/198ac3b7ca087bda FAIL (2026-01-12 14:36:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000586558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X...X...X...X.............X..XXX................] duration: 20.184359719s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:36:15 UTC) orbiter warning: context deadline exceeded Duration: 20.184401809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X............X...X...X...X.............X..XXX...............] duration: 14.463794453s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:35:51 UTC) orbiter warning: context deadline exceeded Duration: 14.463874135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.987842671s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:35:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.987867669s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X.] duration: 28.197233062s trace: https://jaeger.dev.storj.io/trace/29e2e68844fc3c8f FAIL (2026-01-12 14:35:40 UTC) orbiter warning: context deadline exceeded Duration: 28.197284389s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXXX] duration: 5.001662537s trace: https://jaeger.dev.storj.io/trace/6d039189b03bd008 FAIL (2026-01-12 14:35:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00173737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X............X...X...X...X.............X..XXX..............] duration: 28.509181917s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:35:33 UTC) orbiter warning: context deadline exceeded Duration: 28.509203208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.790498679s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.790525759s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X............X...X...X...X.............X..XXX.............] duration: 18.04038177s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:34:59 UTC) orbiter warning: context deadline exceeded Duration: 18.040401317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XXX] duration: 5.001398221s trace: https://jaeger.dev.storj.io/trace/42dc4dc993150486 FAIL (2026-01-12 14:34:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001420923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 10.577060156s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:34:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.577086606s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.809550924s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.809579068s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.X] duration: 30.00049287s trace: https://jaeger.dev.storj.io/trace/29e2e68844fc3c8f FAIL (2026-01-12 14:34:27 UTC) orbiter warning: context deadline exceeded Duration: 30.000503029s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X............X...X...X...X.............X..XXX............] duration: 38.440289473s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:34:18 UTC) orbiter warning: context deadline exceeded Duration: 38.440305192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.XX] duration: 5.00077628s trace: https://jaeger.dev.storj.io/trace/1396fe1b7093a375 FAIL (2026-01-12 14:34:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0007974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX.] duration: 30.001453096s trace: https://jaeger.dev.storj.io/trace/6c554367899f8701 FAIL (2026-01-12 14:33:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001471712s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.X] duration: 5.002572953s trace: https://jaeger.dev.storj.io/trace/ae8ba8a805bdd4c FAIL (2026-01-12 14:33:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002617958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XX..X............X...X...X...X.............X..XXX...........] duration: 30.063723083s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:33:39 UTC) orbiter warning: context deadline exceeded Duration: 30.063751046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.766645481s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:33:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.766669416s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX.] duration: 4.170722608s trace: https://jaeger.dev.storj.io/trace/7df58d2f65b2ad58 FAIL (2026-01-12 14:33:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.170744079s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 10.502785229s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:33:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.50281271s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.774268859s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.77429565s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XXX] duration: 30.001381628s trace: https://jaeger.dev.storj.io/trace/6c554367899f8701 FAIL (2026-01-12 14:33:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001410724s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXXX] duration: 5.001323459s trace: https://jaeger.dev.storj.io/trace/7df58d2f65b2ad58 FAIL (2026-01-12 14:32:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001345933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.7826642s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:32:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.782698726s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.XX..X............X...X...X...X.............X..XXX..........] duration: 20.764057805s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:32:40 UTC) orbiter warning: context deadline exceeded Duration: 20.764092462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 7.079841709s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:32:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.079868279s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..XX] duration: 30.000938367s trace: https://jaeger.dev.storj.io/trace/4df8fcb06619ed85 FAIL (2026-01-12 14:32:16 UTC) orbiter warning: context deadline exceeded Duration: 30.000958947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.845612412s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:31:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.845642329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXXX] duration: 5.000557159s trace: https://jaeger.dev.storj.io/trace/30859cfcc33642cf FAIL (2026-01-12 14:31:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000582828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XXX.X.X..X..X] duration: 30.002517773s trace: https://jaeger.dev.storj.io/trace/750d6eabcf53c0b FAIL (2026-01-12 14:31:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002539013s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..XX..X............X...X...X...X.............X..XXX.........] duration: 34.171760191s trace: https://jaeger.dev.storj.io/trace/2f8d26e9a65e942b FAIL (2026-01-12 14:31:39 UTC) orbiter warning: context deadline exceeded Duration: 34.171790918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.344797022s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:31:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.344828051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.771761054s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:31:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.771812272s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXXX] duration: 5.000954893s trace: https://jaeger.dev.storj.io/trace/6e70cb33aad90d0e FAIL (2026-01-12 14:31:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000966244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [X.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.X.XXX.XX.XXX] duration: 5.001442584s trace: https://jaeger.dev.storj.io/trace/66bd1e339eb65dfc FAIL (2026-01-12 14:30:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001537216s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)