Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.720215025s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:30:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.720240224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.969734055s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.969764302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.691770991s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:29:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.691807872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.607155256s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:28:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.60730889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.821037684s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.821065117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.968190935s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:27:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.968208018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.560504487s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:27:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.560524605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 8.5866029s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.586626395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.075960106s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:26:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.075986377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.793797957s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:26:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.793823084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.852124976s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:25:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.852139894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.679592341s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:25:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.679604063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.788239671s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:24:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.788261232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.169466482s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:24:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.169480118s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.781414717s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:23:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.781439505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.352629006s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:23:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.352654133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.857550386s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.857574924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.490249161s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:22:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.490271633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.090538236s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:22:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.090565168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.24251675s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.242541556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.873703991s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:21:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.873728589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.759976757s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.760006644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.372643703s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:20:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.372669733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.807815487s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:19:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.807853909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 10.046973326s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:19:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.046997162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.552207618s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:19:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.552232383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.185092226s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:19:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.185118786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.62869973s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:18:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.628728276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.696034476s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:18:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.696062419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.786744005s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:17:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.786774114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.91579189s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:16:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.915805425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.757532933s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:16:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.757549294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.761004648s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.76102707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.944814344s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:15:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.944841226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.603986442s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:15:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.604010409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.23806588s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:14:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.238089094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.778553627s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:14:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.77857606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.645951464s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:14:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.645973467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.532084734s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:14:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.532099461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.017157222s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:13:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.017186709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.766618953s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:13:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.766645323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.743403641s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:12:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.74342971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.628771323s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:12:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.628786533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000621934s trace: https://jaeger.dev.storj.io/trace/40719ae419ee0c7b FAIL (2026-05-15 07:12:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000647271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.952911942s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:12:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.952925479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.266638089s trace: https://jaeger.dev.storj.io/trace/6cd842384f67522d FAIL (2026-05-15 07:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 15 07:11:17.133292002 (348h23m27.654115747s) [upload=333776]: Starting upload May 15 07:11:17.133342018 (348h23m27.654165763s) [upload=333776]: Done starting upload May 15 07:11:17.133430116 (348h23m27.654253861s) [upload=333776]: Uploading segments... May 15 07:11:17.133508005 (348h23m27.65433174s) [upload=333776]: Got next segment. Inline: false May 15 07:11:17.133538113 (348h23m27.654361858s) [upload=333776]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 07:11:17.133545096 (348h23m27.654368841s) [upload=333776]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 07:11:17.369169107 (348h23m27.889992842s) [seg_pos={0 0}, upload=333776]: Begin upload segment... May 15 07:11:17.369220355 (348h23m27.89004409s) [seg_pos={0 0}, upload=333776]: Done begin upload segment. May 15 07:11:17.369249861 (348h23m27.890073606s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369263687 (348h23m27.890087432s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369296039 (348h23m27.890119784s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.369680944 (348h23m27.890504689s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370096036 (348h23m27.890919781s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370452266 (348h23m27.891276011s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.370749884 (348h23m27.891573619s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371012925 (348h23m27.89183667s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.371277450 (348h23m27.892101195s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.372979648 (348h23m27.893803403s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.373652191 (348h23m27.894475946s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.375751157 (348h23m27.896574902s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.376520906 (348h23m27.897344651s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.378443495 (348h23m27.89926724s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.379461638 (348h23m27.900285383s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:17.384316512 (348h23m27.905140257s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Uploading piece... May 15 07:11:18.638073351 (348h23m29.158897096s) [upload=333776]: Next returned nil segment May 15 07:11:18.638086084 (348h23m29.158909829s) [upload=333776]: Waiting for error group managing segments... May 15 07:11:18.968734326 (348h23m29.489558071s) [noise=true , address=193.149.164.79:4515 , piece_id=KPPRZ55QRWF2ZJ6NYB4UJCXGJF4HVYUXSSLHC4X76OPJM26SJ4FA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:18.983243504 (348h23m29.504067249s) [noise=true , address=193.149.164.78:4467 , piece_id=2BUULIM5RCKUVODZAPLROP53IDXTHMVTSBBYNUWSHSAISSYIXCUA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.244245672 (348h23m29.765069417s) [noise=true , address=145.79.227.10:4420 , piece_id=ONL5U5W2OBJYYHQONYTUHVKARN2TVF7AF2GEQUBRYL3ICAX4TE2A, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.269673465 (348h23m29.79049721s) [noise=true , address=145.79.227.5:4245 , piece_id=VYXJ7MGQKQ4RQVRFPBNTW4WBYO2SSVKEGZ6T4SPEJOYCLK5MSSHA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273168174 (348h23m29.793991919s) [noise=true , address=145.79.227.51:4423 , piece_id=IEFJCGDCMBBXSH7FFEIHB6JTVNMMR3CSZR3O32JSHV5ZV5RVIPPA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.273708305 (348h23m29.79453205s) [noise=true , address=145.79.227.31:4830 , piece_id=UNRZUCNVQYDEMPLMYIO2IDHMDE3WQXS7NQT43W5XVIZCGCKYVPUQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.307772418 (348h23m29.828596163s) [noise=true , address=145.79.227.48:4325 , piece_id=4WAYBH6MPLWBEQ76LFORGMNKGD5IPW3LXDVUE2PEBQ7QH2NJZFQQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.344740434 (348h23m29.865564179s) [noise=true , address=77.90.31.18:4830 , piece_id=SBL25UCJ64HFUC7JV7VM7RRIWRIGNJGYF7IFA7JT47JN236WQK4Q, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.519444740 (348h23m30.040268486s) [noise=true , address=193.149.164.66:4976 , piece_id=UCO52WNN2OZ76ZH7EBOB7P4SUNKYVGLSBBSSUGMMCASMMSWLJAKA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.709740270 (348h23m30.230564015s) [noise=true , address=5.231.73.3:4005 , piece_id=6WV3MTC6O73KX53L7325RKUXEH5DN5H3XLONXT6SPERCZLJWKUAQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.713723140 (348h23m30.234546885s) [noise=true , address=193.149.164.69:4090 , piece_id=67PYHZH7TM7Q7RSBENSFLN3UPEN2UQ5TZVFLRZWDSPJJXSHPUQNQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:19.918545661 (348h23m30.439369406s) [noise=true , address=45.91.116.12:4237 , piece_id=IIZTIQCD6WZLVTGD4TIBQFR27FANQMRZJHDNHSFBOL5LBHIO2MBA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:11:20.161292751 (348h23m30.682116496s) [noise=true , address=145.79.227.37:4592 , piece_id=DJZEK6NZFQMBQCRCGKL7NHULLI4FGMNND3I5R55Y3GR73AGI4UOA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: <nil> May 15 07:12:16.477485165 (348h24m26.9983089s) [noise=true , address=167.17.72.68:4671 , piece_id=DTQQVGJKQDMFEI2K6QSVGOUXLOUC4KV5ZSQEYQNBMONWV43XYTMQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12NLyT9yuQVGaZYWDPRGyEeL9HhYFT9BH4hmtVjfJQeZQMbHGPc, address:167.17.72.68:4671): protocol: rpc: dial tcp 167.17.72.68:4671: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477769497 (348h24m26.998593242s) [noise=true , address=167.17.72.69:4721 , piece_id=LTCQVH4GHXUSXROKMGHTAHMMYA3NPTVNMBFAFMUI42ITHGM2YTZQ, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: write tcp 188.166.225.170:40086->167.17.72.69:4721: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477901850 (348h24m26.998725595s) [noise=true , address=167.17.72.70:4767 , piece_id=35H75YFSGPC7EWXRH5EUZOSYSPER4MS73P6XMRQSARPY6GMMF3XA, seg_pos={0 0}, upload=333776]: Done uploading piece. err: ecclient: upload failed (node:12GyCnxNSzFpMZgnxBVVKyR6WkY1nrzqQWkuLLpwBri7kWP5rjc, address:167.17.72.70:4767): protocol: rpc: dial tcp 167.17.72.70:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 15 07:12:16.477969980 (348h24m26.998793725s) [seg_pos={0 0}, upload=333776]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 15 07:12:16.742671076 (348h24m27.263494821s) [upload=333776]: Done uploading segments... May 15 07:12:16.742703528 (348h24m27.263527273s) [upload=333776]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.266658369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.625038475s trace: https://jaeger.dev.storj.io/trace/710a707ac7cac304 FAIL (2026-05-15 05:25:18 UTC) orbiter warning: context deadline exceeded Duration: 3.625058183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.89310594s trace: https://jaeger.dev.storj.io/trace/710a707ac7cac304 FAIL (2026-05-15 05:24:40 UTC) orbiter warning: context deadline exceeded Duration: 3.89312185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.87611062s trace: https://jaeger.dev.storj.io/trace/710a707ac7cac304 FAIL (2026-05-15 05:24:05 UTC) orbiter warning: context deadline exceeded Duration: 3.876130919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.772488533s trace: https://jaeger.dev.storj.io/trace/710a707ac7cac304 FAIL (2026-05-15 05:23:09 UTC) orbiter warning: context deadline exceeded Duration: 3.772508652s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)