Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.783625875s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:55:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.783710416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.683088984s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:54:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.683115094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.861929163s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:54:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.861952317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.68328094s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.683328251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.848213832s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.848260161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.871525023s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.871559369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.783058779s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.783089527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.201480231s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:50:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.201503134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.869190139s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:49:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.869224404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.810508266s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:48:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.810548641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.708208436s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:47:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.708238052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.803663252s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.803691877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.777002995s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:46:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.77703248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.754922245s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:46:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.754953514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.065623775s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:45:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.065669772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.803609018s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:43:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.80363611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.765810498s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:43:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.765840706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.858222473s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:42:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.858252279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.777463023s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:42:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.777486857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.832045193s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.832073987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.757101845s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:40:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.757128425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.823964671s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:39:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.823997462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.862714298s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:39:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.862749214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.67196044s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:38:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.671996298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.900709048s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:37:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.900735498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.783170733s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:36:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.783200991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.893118262s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:36:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.893143559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.793634928s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.793667211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.251220676s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 08:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.25124974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.420409359s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420465336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.450455994s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:16:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450487833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.47141945s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:15:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471443126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.74445497s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:14:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.744482833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.56429889s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:13:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564359534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.407305536s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:13:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407338318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.828100206s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:12:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.828131446s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.559602817s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:12:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559615792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.76827505s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.768303725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.686995569s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.687034032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.486726117s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:11:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486754641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.723890923s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:11:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.723917636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.724083368s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.724112484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.367332611s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:10:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36735847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.402472094s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:10:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402518582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.326417684s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:09:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.326437101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.721693669s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:09:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.721726352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.462509827s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462535929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.782839768s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:08:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.7828661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.438912627s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:08:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438942815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.785365084s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.785392146s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)