Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.981573382s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:18:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.981600924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.789919891s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:18:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.789944468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.932388963s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.932417348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.753212073s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.753236138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 11.836185805s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:16:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 11.83620916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.64774055s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:16:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.647769837s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.751314529s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.751477691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.826857548s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.826877206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 22.158750981s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 22.158766941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 24.610858942s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:13:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 24.610880403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.798628553s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.798654081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.82704285s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:12:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.827073959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.887181391s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.88721187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.851565685s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:11:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.851591555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 26.112173863s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:10:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 26.112199974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.75639309s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:09:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.756447594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.683368871s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:08:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 8.683398356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.742113961s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:07:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.74214055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.806054715s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:07:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 5.806084141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002563755s trace: https://jaeger.dev.storj.io/trace/18832a20701264b7 FAIL (2025-12-25 23:06:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: 6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: 6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ) context deadline exceeded; piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A) context deadline exceeded ======LOGS====== Dec 25 23:06:23.457530196 (487h37m23.328613231s) [upload=427975]: Starting upload Dec 25 23:06:23.457608946 (487h37m23.328692011s) [upload=427975]: Done starting upload Dec 25 23:06:23.457791433 (487h37m23.328874478s) [upload=427975]: Uploading segments... Dec 25 23:06:23.457853201 (487h37m23.328936236s) [upload=427975]: Got next segment. Inline: false Dec 25 23:06:23.457877917 (487h37m23.328960952s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 23:06:23.457882136 (487h37m23.328965181s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 23:06:23.706760328 (487h37m23.577843383s) [seg_pos={0 0}, upload=427975]: Begin upload segment... Dec 25 23:06:23.706819480 (487h37m23.577902525s) [seg_pos={0 0}, upload=427975]: Done begin upload segment. Dec 25 23:06:23.706874004 (487h37m23.57795705s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.706907979 (487h37m23.577991014s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707411837 (487h37m23.578494872s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.707790769 (487h37m23.578873804s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708183035 (487h37m23.579266071s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.708572947 (487h37m23.579655982s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709020469 (487h37m23.580103504s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.709404369 (487h37m23.580487424s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.710102619 (487h37m23.581185654s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.711262194 (487h37m23.582345239s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.712771407 (487h37m23.583854452s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.714586040 (487h37m23.585669085s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.716336280 (487h37m23.587419316s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.720320713 (487h37m23.591403748s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.725197503 (487h37m23.596280548s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:23.727021714 (487h37m23.59810477s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Uploading piece... Dec 25 23:06:25.555281665 (487h37m25.42636471s) [upload=427975]: Next returned nil segment Dec 25 23:06:25.555334555 (487h37m25.42641759s) [upload=427975]: Waiting for error group managing segments... Dec 25 23:06:25.916930967 (487h37m25.788014012s) [noise=true , address=145.79.227.50:4374 , piece_id=5AABTCQ3A7A7GFRCMZU3JQRWXY2K4OHU3TZXLWVZA4HNPYSJJ4CA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.918203699 (487h37m25.789286744s) [noise=true , address=145.79.227.51:4411 , piece_id=6FSGQI6ESWMUL2HTTJOVI2C6GNQ74AEGM262IDJUWVB6BGOQ6G3Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.922670749 (487h37m25.793753794s) [noise=true , address=77.90.31.11:4555 , piece_id=S3NWZSLRV7PXJ6GYLXQLZSO4KJZ4GA45JBSXIN3BQNIIYRPHMDUQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.934855576 (487h37m25.805938621s) [noise=true , address=145.79.227.27:4689 , piece_id=T42J5XWVL5W3VWCOKM2EP2M25OTF6EYKIBCT2RVSWZPUUS4VCA6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.938723607 (487h37m25.809806642s) [noise=true , address=145.79.227.38:4979 , piece_id=ZTX7UEBOAD6TMY33DBIEGV63DAPRDO6YBKQJO5B7K4HQ6OPR2LQQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.940766996 (487h37m25.811850051s) [noise=true , address=145.79.227.42:4123 , piece_id=6V2FJX7F7HPSIIXAAWGFFZL5AY2HUAQIOXGG7JMMOFUWK73FW64Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:25.961283317 (487h37m25.832366362s) [noise=true , address=145.79.227.39:4013 , piece_id=POK4AOMDX6EVMEHDIFVMX7VKSAK7WYCXWNEIKP3P3MCKV4TDWQVA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.026964415 (487h37m25.89804746s) [noise=true , address=77.90.31.10:4517 , piece_id=AHN24WXHQGLZCFPDDYXQKRK546PCMN6N3OSKQDHX66SCOXMFX3PQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.040930592 (487h37m25.912013627s) [noise=true , address=45.91.116.8:4045 , piece_id=6F7RMIBHT5MAIRFFBLSGWY73EET3Q6WRWPKHR47KCVZNXDDDGS2Q, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.238688716 (487h37m26.109771751s) [noise=true , address=145.79.227.36:4519 , piece_id=NWITQJPUERMCZLHLQS3HGY3T5JAEUZPS4JGNAS5KSJXGVJT6XDIQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.255590639 (487h37m26.126673684s) [noise=true , address=45.91.116.14:4191 , piece_id=7ZWDXWYDYZ4XWDIZFHCBQDWRRH5VMOZ4MMSO6TGAXEYTZQZC763A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.274128757 (487h37m26.145211802s) [noise=true , address=45.91.116.9:4263 , piece_id=5SHIL6FBQ4LCUZNQNG2EGS5QVPEPOGK3Y53SPZL35M46USZ5RB4A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585476828 (487h37m26.456559863s) [noise=true , address=145.79.227.37:4567 , piece_id=JTSD7BBP7WKA76FB46UKCKIHBSQTVC7YRQAKI67ETMRZU45OQH6A, seg_pos={0 0}, upload=427975]: Done uploading piece. err: <nil> Dec 25 23:06:26.585494601 (487h37m26.456577636s) [seg_pos={0 0}, upload=427975]: Segment reached optimal threshold of 13 pieces. Dec 25 23:06:26.585633906 (487h37m26.456716941s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:46020->77.90.31.21:4877: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585640719 (487h37m26.456723754s) [noise=true , address=77.90.31.21:4877 , piece_id=XOBXDRTQLVAY3HXPBB4ARXWHUT62MXQWKRVJCQPBBX5I7C7WWFJA, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585908760 (487h37m26.456991795s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 25 23:06:26.585915252 (487h37m26.456998297s) [noise=true , address=145.79.227.15:4875 , piece_id=MBCCFBJM3U7Y5WGLYNLP4TXR5IUOZPHBQSMLEEIO6U4IUHOJKWKQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585978112 (487h37m26.457061147s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:42276->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 25 23:06:26.585983352 (487h37m26.457066387s) [noise=true , address=77.90.31.18:4805 , piece_id=GICTEKMVEOS4RZFUHWWRHCJIT3EPOQWVT2GE5WDU5OYI7W6254EQ, seg_pos={0 0}, upload=427975]: Long tail cancellation due to optimal threshold Dec 25 23:06:26.585996917 (487h37m26.457079962s) [seg_pos={0 0}, upload=427975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 23:06:26.586133678 (487h37m26.457216723s) [upload=427975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 23:06:26.586138236 (487h37m26.457221271s) [upload=427975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 23:06:26.886416582 (487h37m26.757499617s) [upload=427975]: Done uploading segments... Dec 25 23:06:26.886441590 (487h37m26.757524625s) [upload=427975]: Upload finished. err: <nil> Duration: 30.002585487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.760532393s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:59:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.760615893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.759862758s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:58:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.759891904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.758777054s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:57:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.758804785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.885096411s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:57:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.885123732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.722809344s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.722835916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.835965461s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:56:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.835998215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.673523056s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:55:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.673549256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.72604171s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.726076104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.798704839s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:54:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.798729939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.849197524s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.84922679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.876229595s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876262507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.745965318s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:52:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.745993652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.669482433s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.66951258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.779795274s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.779821755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.715887985s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:50:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.715913854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.769589842s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:49:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.76961515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.77851276s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.778542886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.743156501s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:47:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.743184265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.86382831s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:47:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.863855802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.772876516s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:46:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.772908438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.70144347s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.701472375s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 9.825583443s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:45:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.825613681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.701732109s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.701747557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.697839163s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:44:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.697866636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.739001948s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:43:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.739027176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 7.109005055s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:42:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.109025313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.692263577s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.692290458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.65760181s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:41:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.657623521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.711367903s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.711394713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.768568663s trace: https://jaeger.dev.storj.io/trace/6ca1efbe5c7087f1 FAIL (2025-12-25 22:40:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 25 22:16:31.901830041 (486h47m31.772913077s) [upload=427256]: Starting upload Dec 25 22:16:31.901942075 (486h47m31.77302514s) [upload=427256]: Done starting upload Dec 25 22:16:31.902011246 (486h47m31.773094301s) [upload=427256]: Uploading segments... Dec 25 22:16:31.902259759 (486h47m31.773342804s) [upload=427256]: Got next segment. Inline: false Dec 25 22:16:31.902305797 (486h47m31.773388862s) [upload=427256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 22:16:31.902311889 (486h47m31.773394954s) [upload=427256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 22:16:32.149883576 (486h47m32.020966612s) [seg_pos={0 0}, upload=427256]: Begin upload segment... Dec 25 22:16:32.149928512 (486h47m32.021011557s) [seg_pos={0 0}, upload=427256]: Done begin upload segment. Dec 25 22:16:32.149960112 (486h47m32.021043147s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150030025 (486h47m32.02111307s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.150767589 (486h47m32.021850624s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151152863 (486h47m32.022235898s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151437185 (486h47m32.02252022s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151729762 (486h47m32.022812797s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.151997873 (486h47m32.023080908s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152241757 (486h47m32.023324792s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152295148 (486h47m32.023378183s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152610980 (486h47m32.023694015s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152617363 (486h47m32.023700398s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152880273 (486h47m32.023963308s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.152989240 (486h47m32.024072285s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153362311 (486h47m32.024445346s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153651040 (486h47m32.024734075s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:32.153954358 (486h47m32.025037393s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Uploading piece... Dec 25 22:16:33.856288883 (486h47m33.727371928s) [upload=427256]: Next returned nil segment Dec 25 22:16:33.856306126 (486h47m33.727389161s) [upload=427256]: Waiting for error group managing segments... Dec 25 22:16:34.165262648 (486h47m34.036345683s) [noise=true , address=77.90.31.14:4663 , piece_id=KFHIRW77LCIJFJERDJQ34ZD4UVSJPA4A6WJLXZY42VTZNPDD3SMQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.195282090 (486h47m34.066365125s) [noise=true , address=45.91.116.9:4264 , piece_id=RRN3QFUYJATIMKLJNR2CM6IJ2CV5POHSFGLXPSULIJTZSJBCZEEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:34.196028010 (486h47m34.067111045s) [noise=true , address=45.91.116.10:4083 , piece_id=BTESIBBLGAC2ZX6VCE64EZIO4DE5DVGKFHSFX5U2QZLWMZBZMOEA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.620423664 (486h47m35.491506699s) [noise=true , address=145.79.227.11:4730 , piece_id=VAUUULQG53HDMUYBDVLT6432WVFO6F2VBF4NOFSTADIHU7LI5MTA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:35.685556967 (486h47m35.556640002s) [noise=true , address=145.79.227.49:4337 , piece_id=NXQB4U74USFZW7DWBJCP74YZXDQHELJ6YE6HEPRRXO2RI52EFIUQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.316140450 (486h47m36.187223485s) [noise=true , address=145.79.227.21:4475 , piece_id=AGUOGASTFWBENIWM3SL2DW63DRPRAKHSOLOQCTHDPRVFU62LFHJA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:36.924511587 (486h47m36.795594622s) [noise=true , address=145.79.227.13:4804 , piece_id=A2TVYXQDKJXPYVUFS5KSA267WVYPHUJHPHZIKNTB2ZNB4IE6WMKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.325984554 (486h47m37.197067589s) [noise=true , address=145.79.227.43:4159 , piece_id=67UVFWZULSV5XKPFO7VORSWBF73CL5ZNYMQHBJK6INYWFEAPLQGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:37.377851947 (486h47m37.248934982s) [noise=true , address=145.79.227.52:4446 , piece_id=RKUNZYY2SFTFGMMM7SOG55H7SDUYXFKVARBYGJETPWA4WQRFUMHQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:40.491428397 (486h47m40.362511442s) [noise=true , address=145.79.227.16:4910 , piece_id=MXHIGCGP67HOGA6EDFFR45ZW5M2K6BH2EQIUTGOKV5GTUNW3IPGA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:46.126992772 (486h47m45.998075807s) [noise=true , address=145.79.227.41:4087 , piece_id=ORWIH624MLMK4CEQBC3WO2P22A35XFUGTINHX3HN2BNYECCG6OKA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:16:59.864735111 (486h47m59.735818156s) [noise=true , address=145.79.227.48:4303 , piece_id=J2KNBOAF3OQJCAGMPNEQXB5Z4FFKOQABAZLTLN2TVIFC4MLZVHVA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: <nil> Dec 25 22:17:01.205114345 (486h48m1.07619738s) [noise=true , address=145.79.227.45:4229 , piece_id=JKADULTCMUOEIQDUE64XUGEPZ5EQU5FHKILRRAML5KMV2QSHCOYQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:145.79.227.45:4229): protocol: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36168->145.79.227.45:4229: use of closed network connection Dec 25 22:17:01.205278678 (486h48m1.076361723s) [noise=true , address=145.79.227.32:4836 , piece_id=DTR6Z4CHEON3BWIKFMJ4RQXX6M6QAOQ6EDR7PGXBKHCE5EVZUXNQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: read tcp 188.166.225.170:41252->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 22:17:01.205338542 (486h48m1.076421587s) [noise=true , address=145.79.227.9:4365 , piece_id=4IIBXAUYLBS2C6SX75XKVKPV5OUU72P4WQS5QK5LTZPTUUMWN7DQ, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50586->145.79.227.9:4365: use of closed network connection Dec 25 22:17:01.205600681 (486h48m1.076683727s) [noise=true , address=145.79.227.15:4876 , piece_id=NI3URIAKZAQI4ZIIKZ7XS2C253G2YIOMJZGTWVCNTTD6RL6EURFA, seg_pos={0 0}, upload=427256]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: context deadline exceeded; context deadline exceeded Dec 25 22:17:01.205652389 (486h48m1.076735424s) [seg_pos={0 0}, upload=427256]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 25 22:17:01.446649119 (486h48m1.317732174s) [upload=427256]: Done uploading segments... Dec 25 22:17:01.446695887 (486h48m1.317778934s) [upload=427256]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.768595325s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)