Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.XX] duration: 30.001669994s trace: https://jaeger.dev.storj.io/trace/16c63fa6e51c1f9a FAIL (2026-01-12 14:21:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001698899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.813011359s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.813038391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX.] duration: 4.26000696s trace: https://jaeger.dev.storj.io/trace/16b37f37ad3a555a FAIL (2026-01-12 14:21:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.260037868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............XX..X............X...X...X...X.............X..] duration: 26.612234213s trace: https://jaeger.dev.storj.io/trace/335bccc966b63521 FAIL (2026-01-12 14:21:05 UTC) orbiter warning: context deadline exceeded Duration: 26.612264921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 18.917218317s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:21:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 18.917244156s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.X] duration: 30.000690347s trace: https://jaeger.dev.storj.io/trace/142885b21124b3bf FAIL (2026-01-12 14:20:52 UTC) orbiter warning: context deadline exceeded Duration: 30.00070188s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X..............XX..X............X...X...X...X.............X.] duration: 51.947901481s trace: https://jaeger.dev.storj.io/trace/335bccc966b63521 FAIL (2026-01-12 14:20:33 UTC) orbiter warning: context deadline exceeded Duration: 51.947918924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXXX] duration: 5.001174663s trace: https://jaeger.dev.storj.io/trace/16b37f37ad3a555a FAIL (2026-01-12 14:20:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001190753s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X.] duration: 29.584109538s trace: https://jaeger.dev.storj.io/trace/60c2bc365f86db72 FAIL (2026-01-12 14:20:22 UTC) orbiter warning: context deadline exceeded Duration: 29.584135216s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.816199084s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:20:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.816227177s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.989479559s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:20:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 9.989505588s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X..............XX..X............X...X...X...X.............X] duration: 1m0.001832713s trace: https://jaeger.dev.storj.io/trace/335bccc966b63521 FAIL (2026-01-12 14:19:41 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001844535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..X] duration: 30.00178578s trace: https://jaeger.dev.storj.io/trace/60c2bc365f86db72 FAIL (2026-01-12 14:19:35 UTC) orbiter warning: context deadline exceeded Duration: 30.001807431s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.142163772s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:19:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.142185853s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXXX] duration: 5.000786446s trace: https://jaeger.dev.storj.io/trace/74b144ded4ccb9c9 FAIL (2026-01-12 14:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000809781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.812347688s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:19:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.812371695s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.926606864s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:19:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.92664179s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXXX] duration: 5.001210283s trace: https://jaeger.dev.storj.io/trace/46f6cdec3a9fb671 FAIL (2026-01-12 14:18:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00123996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 10.844177475s trace: https://jaeger.dev.storj.io/trace/351b474724be2a1d FAIL (2026-01-12 14:18:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.84421082s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.767048498s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.767077632s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.X..............XX..X............X...X...X...X.............] duration: 53.44001471s trace: https://jaeger.dev.storj.io/trace/deb3adabf9d5a90 FAIL (2026-01-12 14:18:40 UTC) orbiter warning: context deadline exceeded Duration: 53.440036772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X..] duration: 24.268711275s trace: https://jaeger.dev.storj.io/trace/1f1278ea194997d FAIL (2026-01-12 14:18:26 UTC) orbiter warning: context deadline exceeded Duration: 24.26873015s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.894347323s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.894378862s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXXX] duration: 5.001316793s trace: https://jaeger.dev.storj.io/trace/38733d9c67f5fbf7 FAIL (2026-01-12 14:18:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001344645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.771282439s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.771320982s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X.] duration: 30.001442646s trace: https://jaeger.dev.storj.io/trace/1f1278ea194997d FAIL (2026-01-12 14:17:48 UTC) orbiter warning: context deadline exceeded Duration: 30.001459868s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXXX] duration: 5.00164447s trace: https://jaeger.dev.storj.io/trace/507a941593cee0e1 FAIL (2026-01-12 14:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001658888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [..XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.X] duration: 30.000674272s trace: https://jaeger.dev.storj.io/trace/1f1278ea194997d FAIL (2026-01-12 14:16:58 UTC) orbiter warning: context deadline exceeded Duration: 30.000764043s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X.X..............XX..X............X...X...X...X............] duration: 21.370082642s trace: https://jaeger.dev.storj.io/trace/deb3adabf9d5a90 FAIL (2026-01-12 14:16:58 UTC) orbiter warning: context deadline exceeded Duration: 21.370101569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.825682318s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.825708367s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XXX] duration: 5.000807225s trace: https://jaeger.dev.storj.io/trace/955597a72f294b4 FAIL (2026-01-12 14:16:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000827384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.98135601s trace: https://jaeger.dev.storj.io/trace/5ec1a2058ce3325 FAIL (2026-01-12 14:16:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.981378642s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X.X..............XX..X............X...X...X...X...........] duration: 10.472986831s trace: https://jaeger.dev.storj.io/trace/deb3adabf9d5a90 FAIL (2026-01-12 14:16:21 UTC) orbiter warning: context deadline exceeded Duration: 10.473013601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [...XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X.] duration: 26.77517527s trace: https://jaeger.dev.storj.io/trace/4341ef36c94ab4ba FAIL (2026-01-12 14:16:04 UTC) orbiter warning: context deadline exceeded Duration: 26.77520235s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.78262146s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:15:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.782650778s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 22.750540605s trace: https://jaeger.dev.storj.io/trace/5ec1a2058ce3325 FAIL (2026-01-12 14:15:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.750559991s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.XX] duration: 5.001276668s trace: https://jaeger.dev.storj.io/trace/5f11bea087a628fb FAIL (2026-01-12 14:15:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001301425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..X.X..............XX..X............X...X...X...X..........] duration: 35.74038305s trace: https://jaeger.dev.storj.io/trace/deb3adabf9d5a90 FAIL (2026-01-12 14:15:34 UTC) orbiter warning: context deadline exceeded Duration: 35.740410351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.767002375s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.767037691s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..X] duration: 30.000787465s trace: https://jaeger.dev.storj.io/trace/4341ef36c94ab4ba FAIL (2026-01-12 14:14:52 UTC) orbiter warning: context deadline exceeded Duration: 30.00081648s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 12.4927427s trace: https://jaeger.dev.storj.io/trace/5ec1a2058ce3325 FAIL (2026-01-12 14:14:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.492770563s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.X] duration: 5.000705591s trace: https://jaeger.dev.storj.io/trace/745868f00d473319 FAIL (2026-01-12 14:14:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000726951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X..X.X..............XX..X............X...X...X...X.........] duration: 8.656531649s trace: https://jaeger.dev.storj.io/trace/deb3adabf9d5a90 FAIL (2026-01-12 14:14:26 UTC) orbiter warning: context deadline exceeded Duration: 8.656558471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 30.000606037s trace: https://jaeger.dev.storj.io/trace/5ec1a2058ce3325 FAIL (2026-01-12 14:14:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000624802s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..] duration: 23.44631507s trace: https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3 FAIL (2026-01-12 14:14:18 UTC) orbiter warning: context deadline exceeded Duration: 23.446331761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.749428663s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.749470512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.] duration: 4.155179845s trace: https://jaeger.dev.storj.io/trace/301c56ec948f5d75 FAIL (2026-01-12 14:14:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.155205856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX.] duration: 26.247074043s trace: https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3 FAIL (2026-01-12 14:13:55 UTC) orbiter warning: context deadline exceeded Duration: 26.247098449s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.855454615s trace: https://jaeger.dev.storj.io/trace/62457ac9390c0b68 FAIL (2026-01-12 14:13:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments... Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment... Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment. Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece... Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments... Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil> Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces. Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments... Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil> Duration: 5.855481125s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX] duration: 5.001605902s trace: https://jaeger.dev.storj.io/trace/301c56ec948f5d75 FAIL (2026-01-12 14:13:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00162046s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)