Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........X.XXX.X.X......X....X...XX.X.XX.......XX.X.X..X...] duration: 52.53427372s trace: https://jaeger.dev.storj.io/trace/7cf4591d633d8777 FAIL (2026-05-15 23:07:48 UTC) orbiter warning: context deadline exceeded Duration: 52.534286103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X....XX...................XX....XXX.......................] duration: 4.153754916s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:07:15 UTC) orbiter warning: context deadline exceeded Duration: 4.153781476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.922939932s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 9.922956553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...........X.XXX.X.X......X....X...XX.X.XX.......XX.X.X..X..] duration: 51.295334005s trace: https://jaeger.dev.storj.io/trace/7cf4591d633d8777 FAIL (2026-05-15 23:06:55 UTC) orbiter warning: context deadline exceeded Duration: 51.295361788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X....XX...................XX....XXX......................] duration: 4.677044157s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:06:45 UTC) orbiter warning: context deadline exceeded Duration: 4.6770669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 8.862095074s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.862154948s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [............X.XXX.X.X......X....X...XX.X.XX.......XX.X.X..X.] duration: 25.005939231s trace: https://jaeger.dev.storj.io/trace/7cf4591d633d8777 FAIL (2026-05-15 23:05:55 UTC) orbiter warning: context deadline exceeded Duration: 25.005963317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X....XX...................XX....XXX.....................] duration: 3.937535943s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:05:52 UTC) orbiter warning: context deadline exceeded Duration: 3.93756106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.599456854s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 6.599481931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.............X.XXX.X.X......X....X...XX.X.XX.......XX.X.X..X] duration: 1m0.000673918s trace: https://jaeger.dev.storj.io/trace/7cf4591d633d8777 FAIL (2026-05-15 23:05:29 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000691272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X....XX...................XX....XXX....................] duration: 4.154052892s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:05:28 UTC) orbiter warning: context deadline exceeded Duration: 4.154073292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.218023039s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:04:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 7.218048768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.....X....XX...................XX....XXX...................] duration: 3.874706033s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:04:38 UTC) orbiter warning: context deadline exceeded Duration: 3.874721844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..............X.XXX.X.X......X....X...XX.X.XX.......XX.X.X..] duration: 28.120827925s trace: https://jaeger.dev.storj.io/trace/2717d18f62fcab1 FAIL (2026-05-15 23:03:56 UTC) orbiter warning: context deadline exceeded Duration: 28.120845309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.154574999s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 6.154600708s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.....X....XX...................XX....XXX..................] duration: 3.841000133s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:03:32 UTC) orbiter warning: context deadline exceeded Duration: 3.841031313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.980125194s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:03:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 7.980144742s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...............X.XXX.X.X......X....X...XX.X.XX.......XX.X.X.] duration: 43.420479164s trace: https://jaeger.dev.storj.io/trace/2717d18f62fcab1 FAIL (2026-05-15 23:02:58 UTC) orbiter warning: context deadline exceeded Duration: 43.420491437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.....X....XX...................XX....XXX.................] duration: 4.251605738s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:02:56 UTC) orbiter warning: context deadline exceeded Duration: 4.251623812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 11.560355117s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:02:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.560377148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [................X.XXX.X.X......X....X...XX.X.XX.......XX.X.X] duration: 1m0.001540836s trace: https://jaeger.dev.storj.io/trace/2717d18f62fcab1 FAIL (2026-05-15 23:02:14 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001563057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.....X....XX...................XX....XXX................] duration: 4.263047938s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:02:13 UTC) orbiter warning: context deadline exceeded Duration: 4.263073907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 8.090497736s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.090527794s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X.....X....XX...................XX....XXX...............] duration: 4.098943003s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:01:51 UTC) orbiter warning: context deadline exceeded Duration: 4.098965475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.605146622s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:00:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 6.605171339s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.................X.XXX.X.X......X....X...XX.X.XX.......XX.X.] duration: 15.512790638s trace: https://jaeger.dev.storj.io/trace/731464dfa58d0317 FAIL (2026-05-15 23:00:43 UTC) orbiter warning: context deadline exceeded Duration: 15.512814916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X.....X....XX...................XX....XXX..............] duration: 4.147739909s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 23:00:38 UTC) orbiter warning: context deadline exceeded Duration: 4.147754676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..................X.XXX.X.X......X....X...XX.X.XX.......XX.X] duration: 1m0.000980131s trace: https://jaeger.dev.storj.io/trace/731464dfa58d0317 FAIL (2026-05-15 23:00:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001004487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.576777948s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 23:00:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 6.576811513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 12.006063116s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:59:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 12.006090327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.....X.....X....XX...................XX....XXX.............] duration: 3.895245626s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:59:26 UTC) orbiter warning: context deadline exceeded Duration: 3.895259252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...................X.XXX.X.X......X....X...XX.X.XX.......XX.] duration: 29.468893601s trace: https://jaeger.dev.storj.io/trace/35a8d99a10e184d4 FAIL (2026-05-15 22:58:56 UTC) orbiter warning: context deadline exceeded Duration: 29.468918579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X.....X.....X....XX...................XX....XXX............] duration: 4.413741049s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:58:55 UTC) orbiter warning: context deadline exceeded Duration: 4.413767449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.529611342s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:58:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 7.529634987s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....................X.XXX.X.X......X....X...XX.X.XX.......XX] duration: 1m0.000718701s trace: https://jaeger.dev.storj.io/trace/35a8d99a10e184d4 FAIL (2026-05-15 22:58:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00072843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 11.545712201s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 11.545736918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X.....X.....X....XX...................XX....XXX...........] duration: 4.197590341s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:58:13 UTC) orbiter warning: context deadline exceeded Duration: 4.197631579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X.....X.....X....XX...................XX....XXX..........] duration: 4.339419361s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:57:40 UTC) orbiter warning: context deadline exceeded Duration: 4.339441874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.....................X.XXX.X.X......X....X...XX.X.XX.......X] duration: 1m0.001704054s trace: https://jaeger.dev.storj.io/trace/161206b6eaed9354 FAIL (2026-05-15 22:57:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001725705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 17.681381638s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:57:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 17.681408718s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X.....X.....X....XX...................XX....XXX.........] duration: 4.203247818s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:57:11 UTC) orbiter warning: context deadline exceeded Duration: 4.203279979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 12.919689473s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:56:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 12.91971494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X.....X.....X....XX...................XX....XXX........] duration: 4.838850908s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:56:10 UTC) orbiter warning: context deadline exceeded Duration: 4.838867049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 8.489563173s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:55:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 8.489577692s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......................X.XXX.X.X......X....X...XX.X.XX.......] duration: 37.975752243s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:55:44 UTC) orbiter warning: context deadline exceeded Duration: 37.97576688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 10.719493258s trace: https://jaeger.dev.storj.io/trace/15d3b92a22e517da FAIL (2026-05-15 22:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 22:28:12.624116703 (363h40m23.144940448s) [upload=348437]: Starting upload May 15 22:28:12.624173291 (363h40m23.144997036s) [upload=348437]: Done starting upload May 15 22:28:12.624244949 (363h40m23.145068694s) [upload=348437]: Uploading segments... May 15 22:28:12.624372873 (363h40m23.145196618s) [upload=348437]: Got next segment. Inline: false May 15 22:28:12.624400435 (363h40m23.14522418s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 22:28:12.624404483 (363h40m23.145228228s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 22:28:12.859773855 (363h40m23.3805976s) [seg_pos={0 0}, upload=348437]: Begin upload segment... May 15 22:28:12.859824852 (363h40m23.380648597s) [seg_pos={0 0}, upload=348437]: Done begin upload segment. May 15 22:28:12.859850081 (363h40m23.380673826s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.859908192 (363h40m23.380731947s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860421581 (363h40m23.381245326s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860676116 (363h40m23.381499871s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.860844147 (363h40m23.381667882s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861069307 (363h40m23.381893052s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861109183 (363h40m23.381932928s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861359691 (363h40m23.382183436s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861420717 (363h40m23.382244462s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861609167 (363h40m23.382432912s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861840148 (363h40m23.382663893s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.861881827 (363h40m23.382705572s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862114271 (363h40m23.382938016s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862194775 (363h40m23.38301852s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862335683 (363h40m23.383159428s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:12.862560993 (363h40m23.383384738s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Uploading piece... May 15 22:28:14.637865390 (363h40m25.158689135s) [upload=348437]: Next returned nil segment May 15 22:28:14.637878394 (363h40m25.158702139s) [upload=348437]: Waiting for error group managing segments... May 15 22:28:14.934709198 (363h40m25.455532933s) [noise=true , address=77.90.31.19:4841 , piece_id=223YF6YZ4NC2LTDEQE56RVYGCM2SRMT2ILWHCQPKT2FOI7OH5TAQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.938255915 (363h40m25.45907966s) [noise=true , address=145.79.227.38:4977 , piece_id=EHJYZXY6SOX376U4K3YX4H5AROJ7U5BL6LCHJPOG6LAI7EWBEXMQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.952898406 (363h40m25.473722151s) [noise=true , address=145.79.227.39:4014 , piece_id=2Z675YSUNI3WO2Q56TQEG2RT7BCSHTZGV4WFTEN3VHHP4ZUVQF3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.966259342 (363h40m25.487083077s) [noise=true , address=77.90.31.15:4700 , piece_id=BUF6AZ7WDSE2KAW5XAIEILPYWIX4TPLRRBEUNFJZSKLDIYOS6TCA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.978548532 (363h40m25.499372277s) [noise=true , address=145.79.227.13:4803 , piece_id=UXZYNRAIQKYT6I3D4MNU5B6TZ4BY3X6O37N3NCVQDXGQRPFJPZRQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982278840 (363h40m25.503102585s) [noise=true , address=77.90.31.18:4807 , piece_id=CFEEF7RQYY3QH7E46TP4I3FLU2NAUPQDKDC55427K26LEGZPNTOA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.982420991 (363h40m25.503244736s) [noise=true , address=145.79.227.26:4653 , piece_id=ORN7PNJ5ZIATTIAWFXFOZTWYWAHPDCK6ACPPZKHJWVRL5JSHVGWQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.995745206 (363h40m25.516568951s) [noise=true , address=145.79.227.42:4121 , piece_id=3BZ7JQPMKM7MQDYIU3DHADJZ2VWO7TZ5FRX74DQDHARU3IX3RAOQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:14.996654011 (363h40m25.517477756s) [noise=true , address=77.90.31.11:4558 , piece_id=G3CWJP5C45RKVFLB7FNX2CXAOUAUXLUJDSLFVLXUQEPBXGNSWEBQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.082355686 (363h40m25.603179431s) [noise=true , address=145.79.227.7:4296 , piece_id=HFSKNFFXDEYALYT62BYTLZWV2BXUDZBAQRUJ3F7ARHOD57QEE7RQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.092233315 (363h40m25.61305706s) [noise=true , address=145.79.227.50:4373 , piece_id=Z2DG3MQUGAVDROCN65PQJXFGZEIAOW7KXSH7LXTOGG5YI3MZIZ6A, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.124769251 (363h40m25.645592996s) [noise=true , address=145.79.227.25:4622 , piece_id=5Z6JRT3N4AXV3AFK73H73HKEGQHEXZ5ICIKLS7OFXKKR2SYUWSEA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596706752 (363h40m26.117530497s) [noise=true , address=45.91.116.4:4947 , piece_id=KNG5P7C2EGWMBFM4TVIULRHSOOQUWOLPAGERVA7ZVXMZDWYB7LYQ, seg_pos={0 0}, upload=348437]: Done uploading piece. err: <nil> May 15 22:28:15.596727262 (363h40m26.117551007s) [seg_pos={0 0}, upload=348437]: Satisfied cohort requirements after 13 pieces May 15 22:28:15.596885062 (363h40m26.117708808s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42910->145.79.227.4:4187: use of closed network connection May 15 22:28:15.596890423 (363h40m26.117714168s) [noise=true , address=145.79.227.4:4187 , piece_id=HBGVIH4GLFZRDII7RXFZK4I4K32PPA7QPP446LKZTM7V34PWA5LA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597074043 (363h40m26.117897788s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597079663 (363h40m26.117903408s) [noise=true , address=145.79.227.15:4878 , piece_id=E7BK3E7KZKYMCS6P57WAUVC54FCHZNDUHLEEIYWLUCV5B2WDYN3Q, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597101896 (363h40m26.117925701s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1): protocol: expected piece hash; context canceled; EOF May 15 22:28:15.597120802 (363h40m26.117944548s) [noise=true , address=145.79.227.8:4334 , piece_id=HHBNK6LESFHRKJDGPXEWWACYVNRXUKURQD5AAKHRLBDWXWXMOWHA, seg_pos={0 0}, upload=348437]: Long tail cancellation due to optimal threshold May 15 22:28:15.597155839 (363h40m26.117979584s) [seg_pos={0 0}, upload=348437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 22:28:15.597220392 (363h40m26.118044137s) [upload=348437]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 22:28:15.597224660 (363h40m26.118048405s) [upload=348437]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 22:28:15.885904640 (363h40m26.406728395s) [upload=348437]: Done uploading segments... May 15 22:28:15.885925369 (363h40m26.406749114s) [upload=348437]: Upload finished. err: <nil> Duration: 10.719516784s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X.....X.....X....XX...................XX....XXX.......] duration: 5.001349951s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:55:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001379146s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......................X.XXX.X.X......X....X...XX.X.XX......] duration: 35.518956156s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:55:06 UTC) orbiter warning: context deadline exceeded Duration: 35.518986414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X.....X.....X....XX...................XX....XXX......] duration: 3.894319509s trace: https://jaeger.dev.storj.io/trace/434e3e956d043909 FAIL (2026-05-15 22:54:54 UTC) orbiter warning: context deadline exceeded Duration: 3.894374024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........................X.XXX.X.X......X....X...XX.X.XX.....] duration: 19.421647213s trace: https://jaeger.dev.storj.io/trace/5245a61f6766f81d FAIL (2026-05-15 22:54:28 UTC) orbiter warning: context deadline exceeded Duration: 19.421669255s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)