Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X.....X.....X....XX...................XX....XXX......] duration: 3.894319509s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:54:54 UTC) orbiter warning: context deadline exceeded Duration: 3.894374024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........................X.XXX.X.X......X....X...XX.X.XX.....] duration: 19.421647213s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:54:28 UTC) orbiter warning: context deadline exceeded Duration: 19.421669255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 11.406283659s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.406306672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.......X.....X.....X....XX...................XX....XXX.....] duration: 3.783797068s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:54:09 UTC) orbiter warning: context deadline exceeded Duration: 3.783825983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........................X.XXX.X.X......X....X...XX.X.XX....] duration: 31.346685194s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:53:55 UTC) orbiter warning: context deadline exceeded Duration: 31.346736503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 11.499780412s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:53:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.499829887s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.......X.....X.....X....XX...................XX....XXX....] duration: 4.101862091s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:53:37 UTC) orbiter warning: context deadline exceeded Duration: 4.101874755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 8.429410539s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.429433923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........................X.XXX.X.X......X....X...XX.X.XX...] duration: 43.016966834s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:53:08 UTC) orbiter warning: context deadline exceeded Duration: 43.017005477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X.......X.....X.....X....XX...................XX....XXX...] duration: 3.878520199s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:53:00 UTC) orbiter warning: context deadline exceeded Duration: 3.878577158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..........................X.XXX.X.X......X....X...XX.X.XX..] duration: 18.563385136s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:52:08 UTC) orbiter warning: context deadline exceeded Duration: 18.563403893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.641732965s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:52:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 9.641757772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X.......X.....X.....X....XX...................XX....XXX..] duration: 3.930295896s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:51:53 UTC) orbiter warning: context deadline exceeded Duration: 3.930320643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 11.084926827s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.084956133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.675269495s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:50:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 9.675299743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [....X.......X.....X.....X....XX...................XX....XXX.] duration: 4.570751191s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:50:47 UTC) orbiter warning: context deadline exceeded Duration: 4.570777831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX..........................X.XXX.X.X......X....X...XX.X.XX.] duration: 8.605515284s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:50:47 UTC) orbiter warning: context deadline exceeded Duration: 8.60557589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX..........................X.XXX.X.X......X....X...XX.X.XX] duration: 1m0.001379802s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:50:38 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001393257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 27.172230219s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:50:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 27.172255337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X.......X.....X.....X....XX...................XX....XXX] duration: 5.001189263s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:49:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001199662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..XX..........................X.XXX.X.X......X....X...XX.X.X] duration: 1m0.000990541s trace: https://jaeger.dev.storj.io/trace/3a6f9cd2ddbe465b FAIL (2026-05-15 22:49:37 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00101133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 21.975720308s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:49:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 21.975746819s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X.......X.....X.....X....XX...................XX....XX] duration: 5.001876927s trace: https://jaeger.dev.storj.io/trace/74d8f15ffc222287 FAIL (2026-05-15 22:49:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001897886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X.......X.....X.....X....XX...................XX....X] duration: 5.001231392s trace: https://jaeger.dev.storj.io/trace/65d599f14caa017b FAIL (2026-05-15 22:48:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001244247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 9.901814175s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:48:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 9.901841199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........X.......X.....X.....X....XX...................XX....] duration: 3.803172339s trace: https://jaeger.dev.storj.io/trace/6893f3d7f38420b8 FAIL (2026-05-15 22:48:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80320503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..XX..........................X.XXX.X.X......X....X...XX.X.] duration: 25.005404151s trace: https://jaeger.dev.storj.io/trace/1e1dc50650646ded FAIL (2026-05-15 22:48:07 UTC) orbiter warning: context deadline exceeded Duration: 25.005415883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX..XX..........................X.XXX.X.X......X....X...XX.X] duration: 1m0.00116496s trace: https://jaeger.dev.storj.io/trace/1e1dc50650646ded FAIL (2026-05-15 22:47:42 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001175269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 8.377221111s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:47:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.377243944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........X.......X.....X.....X....XX...................XX...] duration: 4.50436514s trace: https://jaeger.dev.storj.io/trace/6893f3d7f38420b8 FAIL (2026-05-15 22:47:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.504387802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.XX..XX..........................X.XXX.X.X......X....X...XX.] duration: 58.059527019s trace: https://jaeger.dev.storj.io/trace/419091e4ed8e2f3a FAIL (2026-05-15 22:46:41 UTC) orbiter warning: context deadline exceeded Duration: 58.059538943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.609588095s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:46:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 6.609609797s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........X.......X.....X.....X....XX...................XX..] duration: 3.944043922s trace: https://jaeger.dev.storj.io/trace/6893f3d7f38420b8 FAIL (2026-05-15 22:46:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.944070263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........X.......X.....X.....X....XX...................XX.] duration: 3.884673251s trace: https://jaeger.dev.storj.io/trace/6893f3d7f38420b8 FAIL (2026-05-15 22:45:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.884696745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..XX..XX..........................X.XXX.X.X......X....X...XX] duration: 1m0.001137581s trace: https://jaeger.dev.storj.io/trace/419091e4ed8e2f3a FAIL (2026-05-15 22:45:43 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001146759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 10.789243987s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:45:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 10.789262452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 21.846318011s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 21.846346735s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [............X.......X.....X.....X....XX...................XX] duration: 5.001722419s trace: https://jaeger.dev.storj.io/trace/6893f3d7f38420b8 FAIL (2026-05-15 22:45:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001735474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 25.008280525s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:44:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 25.008305554s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............X.......X.....X.....X....XX...................X] duration: 5.001011671s trace: https://jaeger.dev.storj.io/trace/71bbfbb097ef87e9 FAIL (2026-05-15 22:44:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001023424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...XX..XX..........................X.XXX.X.X......X....X...X] duration: 1m0.001041704s trace: https://jaeger.dev.storj.io/trace/2c5db05976c01493 FAIL (2026-05-15 22:44:43 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001053085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.............X.......X.....X.....X....XX...................] duration: 4.879032421s trace: https://jaeger.dev.storj.io/trace/1cb4c90b9422be55 FAIL (2026-05-15 22:44:22 UTC) orbiter warning: context deadline exceeded Duration: 4.879053391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.319476173s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.319502573s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X...XX..XX..........................X.XXX.X.X......X....X...] duration: 58.451749847s trace: https://jaeger.dev.storj.io/trace/7439900a15bb5b43 FAIL (2026-05-15 22:43:43 UTC) orbiter warning: context deadline exceeded Duration: 58.451775695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.............X.......X.....X.....X....XX..................] duration: 3.827962785s trace: https://jaeger.dev.storj.io/trace/1cb4c90b9422be55 FAIL (2026-05-15 22:43:29 UTC) orbiter warning: context deadline exceeded Duration: 3.827985048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.............X.......X.....X.....X....XX.................] duration: 5.001499351s trace: https://jaeger.dev.storj.io/trace/1cb4c90b9422be55 FAIL (2026-05-15 22:42:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001521694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 8.646637338s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:42:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.646657225s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XX...XX..XX..........................X.XXX.X.X......X....X..] duration: 8.250632748s trace: https://jaeger.dev.storj.io/trace/7439900a15bb5b43 FAIL (2026-05-15 22:42:34 UTC) orbiter warning: context deadline exceeded Duration: 8.250644872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 9.027173259s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:42:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 9.027195701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.............X.......X.....X.....X....XX................] duration: 3.848826817s trace: https://jaeger.dev.storj.io/trace/1cb4c90b9422be55 FAIL (2026-05-15 22:41:50 UTC) orbiter warning: context deadline exceeded Duration: 3.84889135s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)