Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.862690199s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:24:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.862713734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....................X.X............X.....X...........XX....] duration: 4.498415761s trace: https://jaeger.dev.storj.io/trace/447b50edfb9ed696 FAIL (2026-05-10 14:24:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.498460345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.93870776s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:24:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.938733038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.882925452s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:24:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.882950109s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................X.X............X.....X...........XX...] duration: 3.705341726s trace: https://jaeger.dev.storj.io/trace/447b50edfb9ed696 FAIL (2026-05-10 14:24:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.70535417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.839389708s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:23:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.839452869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.427206293s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.427229037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......................X.X............X.....X...........XX..] duration: 4.207906701s trace: https://jaeger.dev.storj.io/trace/447b50edfb9ed696 FAIL (2026-05-10 14:23:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.207931117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.766019235s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:23:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.766033303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.6790073s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.679029864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.716589219s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:22:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.716608997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........................X.X............X.....X...........XX.] duration: 4.864906341s trace: https://jaeger.dev.storj.io/trace/447b50edfb9ed696 FAIL (2026-05-10 14:22:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.864934645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.772541335s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.772568669s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.845475592s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:21:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.845505169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.749835247s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:21:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.749869362s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........................X.X............X.....X...........XX] duration: 5.001393861s trace: https://jaeger.dev.storj.io/trace/447b50edfb9ed696 FAIL (2026-05-10 14:21:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001405673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.330857938s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.330877254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........................X.X............X.....X...........X] duration: 5.000659896s trace: https://jaeger.dev.storj.io/trace/385a5b2b34dffd0d FAIL (2026-05-10 14:20:59 UTC) orbiter warning: context deadline exceeded Duration: 5.000678703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.844972112s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:20:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.844992813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.011249566s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:20:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.0112732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.792703983s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:20:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.792727548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.289324865s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:20:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.28935358s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X.X............X.....X...........] duration: 3.715953528s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:19:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.715974087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.054011188s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 7.054037448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.533609541s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:19:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.533637465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.793408369s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:19:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.793436211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.731876089s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:18:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.731899743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.065332335s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:18:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.065355779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X.X............X.....X..........] duration: 3.561848235s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:18:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561874825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.742537244s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:18:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.742559987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.069856627s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 7.069885272s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X.X............X.....X.........] duration: 3.607161152s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:17:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607184877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.198334376s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.198358071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.541733748s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:17:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.541752684s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................X.X............X.....X........] duration: 3.759922843s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:16:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.75994656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.637055057s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.637080605s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.43548911s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.435511071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.61911148s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:16:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.619127342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................X.X............X.....X.......] duration: 3.811575084s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:15:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.811597215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.481136633s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 6.481162722s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.032983509s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:15:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.033050708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.678156818s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:15:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.678182968s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................X.X............X.....X......] duration: 4.253325285s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:15:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.2533443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.79302358s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.793216268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.957359023s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:14:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.957384531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.806190936s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.80621369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................X.X............X.....X.....] duration: 4.57287887s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:14:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.572897896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.579139341s trace: https://jaeger.dev.storj.io/trace/6648ff2bad076ed9 FAIL (2026-05-10 14:13:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jzasZMDY1SsvNgPbDJ2LqhBiHbLnGgHcnXEkUZ4a6UDoCpgsT, Piece ID: OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA) context deadline exceeded ======LOGS====== 21893s) [upload=225259]: Uploading segments... May 10 14:01:16.224858523 (235h13m26.745682258s) [upload=225259]: Got next segment. Inline: false May 10 14:01:16.224887649 (235h13m26.745711404s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 10 14:01:16.224892628 (235h13m26.745716373s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 10 14:01:16.458648925 (235h13m26.97947267s) [seg_pos={0 0}, upload=225259]: Begin upload segment... May 10 14:01:16.458694282 (235h13m26.979518027s) [seg_pos={0 0}, upload=225259]: Done begin upload segment. May 10 14:01:16.458731002 (235h13m26.979554777s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458759456 (235h13m26.979583202s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.458843968 (235h13m26.979667713s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459127068 (235h13m26.979950813s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459464842 (235h13m26.980288587s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.459745368 (235h13m26.980569113s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460016295 (235h13m26.98084004s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460259599 (235h13m26.981083344s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.460587765 (235h13m26.98141151s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.462023354 (235h13m26.982847109s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.463113816 (235h13m26.983937561s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.464084608 (235h13m26.984908353s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.465045181 (235h13m26.985868926s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.466126985 (235h13m26.98695073s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467130310 (235h13m26.987954055s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:16.467480458 (235h13m26.988304203s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Uploading piece... May 10 14:01:18.171956868 (235h13m28.692780613s) [upload=225259]: Next returned nil segment May 10 14:01:18.172017063 (235h13m28.692840808s) [upload=225259]: Waiting for error group managing segments... May 10 14:01:18.458161229 (235h13m28.978984994s) [noise=true , address=145.79.227.39:4016 , piece_id=WYNFXLW5YHTQ77GFNVG6DSSNHM65TCLDZLHMHN24M4TJ72J77L7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.463350482 (235h13m28.984174227s) [noise=true , address=145.79.227.5:4227 , piece_id=IVPLAT5U3FCK5ZE4H635SOILFPPL3BVARLL2X4V6DLFDFO6RFIXA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.465465377 (235h13m28.986289122s) [noise=true , address=145.79.227.19:4077 , piece_id=D7FZF5U4CDSTLAEZZJSOGMYKKMHCEZETWPPUPYB6IS4MCZAJKI4Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.467489280 (235h13m28.988313025s) [noise=true , address=145.79.227.7:4295 , piece_id=HMS5EPXF56BWO3LSSC2E7APE6Q5EOQIZPOMN3I3BWIAHT2GTRAGQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.470505204 (235h13m28.991328949s) [noise=true , address=145.79.227.36:4521 , piece_id=I4364ZTQCMJ3OKC244CYNCE7FTWUFMAVFC5D5HPD7WXV6X47A5FA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.476761352 (235h13m28.997585097s) [noise=true , address=145.79.227.35:4483 , piece_id=3XXB2W24ZTPROFOQSXVXDWVZGBOBO27BXXNPHR2435V22DUT4OGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.479218581 (235h13m29.000042326s) [noise=true , address=145.79.227.47:4269 , piece_id=H2VL3PJ4DRT6WRDH5IENVPPHCIUSW7BSDWMC5Q2XXNTGFHB6E5BA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.487335090 (235h13m29.008158855s) [noise=true , address=145.79.227.51:4409 , piece_id=AQH33YG66KJSTKN7UKB7VL63TGTLYH7PBHUBGNL2VEBURWRUJMZQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.498784128 (235h13m29.019607873s) [noise=true , address=145.79.227.24:4587 , piece_id=SPOSDKPJDIEPTVJEWKEPMFXPSYH4GDB5WGRVEGTB2QEKX7JGXJ5Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:18.504268754 (235h13m29.025092499s) [noise=true , address=77.90.31.19:4845 , piece_id=B765JJPUU3FV5AMK66WZ3IJ7RY4QZJBG6XT3CMUOK3QPEAKUPJUQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:19.367580157 (235h13m29.888403903s) [noise=true , address=145.79.227.9:4369 , piece_id=I4SQRRMM5T4BALW5SLKVSNKFRJ2RVBNWRA655DJC4II72SHGTAGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:24.805735031 (235h13m35.326558766s) [noise=true , address=45.91.116.4:4953 , piece_id=OPUMX2OD5BXLVEMJJ4BDX6WK75TC42LNWJO5NBB45OP446ETJL7A, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992630767 (235h13m46.513454532s) [noise=true , address=45.91.116.7:4012 , piece_id=T6U2R7PBHYMNU5SSQGSJWDOFRKQESQIJDIKRSHP6IP2VMFX7TYGA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: <nil> May 10 14:01:35.992651637 (235h13m46.513475392s) [seg_pos={0 0}, upload=225259]: Satisfied cohort requirements after 13 pieces May 10 14:01:35.993179815 (235h13m46.51400356s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54904->45.91.116.6:4989: use of closed network connection May 10 14:01:35.993188822 (235h13m46.514012567s) [noise=true , address=45.91.116.6:4989 , piece_id=MTOT7ZFNET4CGHUQ4HZTNMR5GLFVYL2ALW4AFRHHJCMLBJKLX66Q, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993412679 (235h13m46.514236424s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:44272->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993422178 (235h13m46.514245933s) [noise=true , address=45.91.116.5:4299 , piece_id=2PF4DOX24VUNBWJPCS6SONGUDLL27G6EH5ASHZT67NSB3BA4VCPQ, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993607952 (235h13m46.514431697s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ): protocol: read tcp 188.166.225.170:57672->45.91.116.8:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 10 14:01:35.993615226 (235h13m46.514438971s) [noise=true , address=45.91.116.8:4049 , piece_id=UV6YT64LYO3U724IMPOGGWL2XCTQ4DLUUBAJ5MDI5KR4BLM2JQRA, seg_pos={0 0}, upload=225259]: Long tail cancellation due to optimal threshold May 10 14:01:35.993648048 (235h13m46.514471793s) [seg_pos={0 0}, upload=225259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 10 14:01:35.993741707 (235h13m46.514565452s) [upload=225259]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 10 14:01:35.993749172 (235h13m46.514572917s) [upload=225259]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 10 14:01:36.289004295 (235h13m46.809828041s) [upload=225259]: Done uploading segments... May 10 14:01:36.289022049 (235h13m46.809845794s) [upload=225259]: Upload finished. err: <nil> Duration: 5.579161685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.072628158s trace: https://jaeger.dev.storj.io/trace/57e77812227844e8 FAIL (2026-05-10 14:13:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.072697851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................X.X............X.....X....] duration: 4.016766283s trace: https://jaeger.dev.storj.io/trace/35caab641616e4aa FAIL (2026-05-10 14:13:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.016787563s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)